Well to start with people running small capacity strategies tend to be just as secretive as running a large capacity strategy. Strategy capacity is something that professionals talk about and doesn't get as much play when talking to a retail audience.
As for the approach, its not any different than finding a high capacity strategy. It requires some piece of information or insight that other market participants don't have. Consider a scalping strategy that trades a few different futures contracts. If on average we trade 200 times per day with an expected profit of $5 per trade with are making $5,000 per week with our strategy. If we say we spend $5,000 per month on the tech to run our business (a risk system, market data, compute time, etc...) we are making $15,000 of profit each month.
If we are a large hedge fund or prop operation the $15,000 per month (assuming the same costs) may or may not be worth running. As a trader say I am making $250K base plus some percentage of my P&L I would definitely need to be running more than that strategy to justify my job. Depending on how much attention it requires it might not be worth the company running it. If I have two other strategies that each make $100,000 per month for the business am I better of in investing in those strategies or one that makes a lot less money? The answer could be yes (like maybe I could add a hundred more instruments to trade) but just like any other business the investment will be evaluated versus the expected returns of my other options.
I understand the last paragraph, my question is more: what makes strategies not scale. Should you go looking in very niche sectors, where the transaction volume limits the scalability? Or are there particular classes of market insights that by their nature only allow (relatively) small profits? Or are there areas where one can exploit particular technical skills? Or maybe focus on geographically small areas somehow, which by their nature limit the potential profit and thus might keep out big players?
I'm not a trader and the level of my questions probably shows that; still despite my (amateur) research for years, I haven't found evidence of people successfully deploying such strategies. And while particular strategies of big players are secret, there is a lot of information on the general principles; for small setups, nothing (afaik). So that leads me more and more to the conclusion that it's simply not viable.
Looking at it another way: a trader who got his experience in a big fund, and who goes solo (a documented scenario), do they go after such inherently small markets, or do they do the same they'd do at a big fund only with less money or with less sophistication? In other words, if they'd have more money, could they scale up or not?
So if you think about any opportunity in the market you are bounded by how much liquidity is available at a price level that is mispriced. In many cases the size (or capacity) is correlated pretty closely this what the expected duration of a trade is and the size of the mis-pricing. In my example of a futures contract that is not very liquid as you move into the market and buy contracts you will naturally move the price such that the price moves to the level you were expecting it to. This limits how much capital can be deployed in an single trade.
I run a strategy currently which consistently is profitable. I know others that do as well. What I run currently is work that came out of starting an automated trading shop so my partner and I have a considerable amount of infrastructure at our disposal that others just starting might not. I currently work elsewhere in finance but may return to it full time when what I am working on now either succeeds or fails. There have been a proliferation of 2 - 5 person shops that are typically pretty secretive about what they do. Several people I know in different ones don't even say that they have a job on linked in.
That being said there is a lot more available off the shelf things available now (Quantopian,etc...) then there was 5 years ago when I tried.
To your last point going to different markets is not just something that individuals can do. There are for instance HFT firms that started trading in places like brazil given the competitiveness of US markets. Markets are only long run efficient.
As for the approach, its not any different than finding a high capacity strategy. It requires some piece of information or insight that other market participants don't have. Consider a scalping strategy that trades a few different futures contracts. If on average we trade 200 times per day with an expected profit of $5 per trade with are making $5,000 per week with our strategy. If we say we spend $5,000 per month on the tech to run our business (a risk system, market data, compute time, etc...) we are making $15,000 of profit each month.
If we are a large hedge fund or prop operation the $15,000 per month (assuming the same costs) may or may not be worth running. As a trader say I am making $250K base plus some percentage of my P&L I would definitely need to be running more than that strategy to justify my job. Depending on how much attention it requires it might not be worth the company running it. If I have two other strategies that each make $100,000 per month for the business am I better of in investing in those strategies or one that makes a lot less money? The answer could be yes (like maybe I could add a hundred more instruments to trade) but just like any other business the investment will be evaluated versus the expected returns of my other options.