If you have been working out for a steady amount of time and consistency then you will not see any sort of losses for a few weeks but even then its more of a over time process and one that you will have trouble noticing unless you have some before after images to compare what you may have looked like a few weeks prior to not exercising or working out. There isnt a whole lot of evidence on how fitness continues to decrease after a period of time but it would appear that it takes a few months in order for individuals to lose most of their aerobic fitness progress. This also varies person to person because each individual has different capabilities. Also this stat is only true if there is no activity at all during this period, most people just take breaks and even then it rarely means absolutely zero activity.
Strength training works a little differently, studies have shown that the older you are the faster the muscles will decrease in mass but strength loss is much slower than any form of aerobic abilities lost during a certain time period. One study shows that after 31 weeks of no weight training young adults lost only 8 percent of the strength gained over 9 weeks before the break period, while older adults lost 14 percent over the same time period.
While this may put some people off the idea of taking breaks, they are a good method of recovering from a harsh set or training, or any rigorous activity. The recovering/healing period after fitness is arguably just as important as the actual exercise as your muscles grow during the healing process and thats how you get stronger.