StoryMill 3 and StoryMill 4 use a different file format.  StoryMill 3 files end in ".storymill' or '.avenirproj. and StoryMill 4 files end in '.smdb'

You can't open a StoryMill 3 file in StoryMill 4, but you can easily import the file:
- Click on the File Menu in StoryMill 4
- Move your mouse to the Import option
- Choose "StoryMill 3.x file..." from the submenu
- Locate the StoryMill 3 file you want to open in StoryMill 4
- Click on that file and click Open

Your StoryMill 3 file will now be imported into a new, unsaved StoryMill 4 file.  You should save the file at this point.

Note that StoryMill 4 files cannot be opened with StoryMill 3.