lzx_compress: make lzx_cost_for_probability() more robust