The routing has to be:

BNC Wordclock cable OUT of the Rosetta 800
to the Wordclock INPUT on the Firewire1814

Optical cable plugged into the Apogee Rosetta800 SMUX out
to the Firewire1814 Adat IN
Also Optical cable from the 1814 Adat out to the Apogee
Rosetta SMUX Input

Settings on the front panel of the Apogee need to be:
Sample Rate to 96Khz
Source to Digital Outs: Analog
Source to Analog Out: ADAT and SMUX

In the Firewire1814 Control Panel, Hardware Tab under
Digital Port Selection, choose ADAT and make sure to
click on the adat(S/MUX) button.
Sync source needs to be set to WORD CLOCK