With the ever increasing complexity of models  used in modern science, there is a need for new computing strategies. Classical MCMC algorithms (Metropolis-Hastings, Gibbs) have difficulty handling very high-dimensional state spaces and models where likelihood evaluation is impossible. In this work we study a collection of models for which the likelihood cannot be evaluated exactly; however, it can be estimated unbiasedly  in an efficient way via distributed computing.  Such models include, but are not limited to cases where the data are discrete noisy observations from a class of diffusion processes or partial measurements of a solution to a partial differential equation. In each case, an exact MCMC algorithm targeting the correct posterior distribution can be obtained either via the ``auxiliary variable trick'' or by using a Bernoulli factory to advance the current state.  We explore the advantages and disadvantages of such MCMC algorithms and show how they can be used in an application from oceanography.