This was probably one of my classes in my Masters program, but most of it was learned from hearing people starting the same nonprofit over and over.
Step One: Define Your Mission
Your mission is what your nonprofit will do.
Now, think about a nonprofit as a business for a minute or two.
Is your mission important (impacts many people) or important to you? If you can't find board members or people give you a blank look when you explain your mission, maybe it's important to you. That's fine, it just may not be a nonprofit. Write a book and self-publish it.
How many other nonprofits in the area has the same or a similar mission?
How many for-profits in the area provide similar services?
How are they doing financially? (Form 990s are useful for this.)
How long have they been around?
How many clients do they serve? (Look at their website/newsletters.)
Does the government provide similar services?
If you are starting a nonprofit to help feed people, and there is a million-dollar annual budgeted, twenty-year old, well-respected food bank down the street, why not just volunteer there and avoid all the paperwork of doing your own nonprofit?
Who is your clientele? (Define by Age, Sex, Marital Status, Income, etc)
How many potential clients are in your service area? (www.census.gov can help)
Is the number of clients consistent? Growing? Shrinking?
How will you find your clients?
How will they find you?
If you are starting a nonprofit to help lower-income homeowners in a neighborhood that is mostly upscale apartments, that's not a good area to serve.
If you are starting a nonprofit to help elders learn to use the Internet, just putting up a website will probably not be enough - if they could find your website, they don't need your services!