It does seem a little weird not to see them all the time, but I suppose the only time you really need them is when you are editing a formula, and they will appear when you are in the formula editor.
Double tap on a cell, then choose the = sign from editor choices. Columns and rows will be labeled.
I looked for another way, and was almost certain there was because I'd seen them before, but in retrospect I'm guessing it was when I was editing formulas, which would be the only time I really cared.
If you need a numbered row you can create a header row and auto-fill it. Enter the first couple numbers with the right interval (1,2 / 1,3 / 1, 1.25 / etc) tap to select the first cell, drag to select all the cells with data, tap and hold for the popup menu, choose fill, then drag the result to fill all the remaining cells. Works with dates too.