Unfortunately, just about everything is wrong with this process. Workers are called in when they aren't ready to work, and they have little to no training or experience in the job they're about to do. They're counting your inventory on their day off, they don't know what they're doing, and the job doesn't get done. Half the time, you make adjustments to make the numbers line up, which actually mess up the numbers that are correct. If you're in operations careers, this is a huge mistake. Yet people in operations management careers do it all the time.
What's the answer? Don't take inventory once a year. Do it more often, in smaller chunks, and plan for it. This simple change can make all the difference. People in operations management careers need to take the time to think hard about how they're going to deal with inventory, not make it the last consideration. Counting items on a regular basis throughout the year is called cycle counting. Many people think of this as a daily product counting, with some random element. However, this doesn't have to be the only definition of this method in operations careers.
Cycle counting can also be thought of as any count program that makes use of regularly scheduled counts involving less than the whole inventory of the facility. One excellent method is to hybridize the usual physical inventory method with regular cycle counting - count everything in a given area, just like a physical inventory, but don't do the whole warehouse at once. Start where you left off last time when you take up the next cycle. Soon, everything will be counted, your staff will have a good idea of how to do it, and you'll have a much more accurate count.
That doesn't mean a regular physical inventory doesn't work for some. If your operation is small, the counters are well trained and have plenty of time, and there's time to check out discrepancies before you make an adjustment, you can be successful in this. However, for anyone who has an extensive inventory, cycle counting is just a better idea. There's no reason to do physical inventory once a year, inconvenience everyone, and get an incorrect result when you can simply do your counting over a longer period of time and get better results. It's that simple.