"Ultimately" is a word that you use when you want to make a conclusion. Use it to summarize a decision or a final point that you want to make:
There are a lot of different options. Ultimately, though, it comes down to which one is cheapest.
You can also use "ultimately" to talk about something that happens eventually, after a long period of time:
The popular theory these days is that some dinosaurs ultimately evolved into birds.
You can see from the examples that "ultimately" can come at the beginning of a sentence, or it can come in the middle before a verb.