We extend the penalty immersed boundary (pIB) method to investigate the interaction between circular rigid particles and a surrounding viscoelastic Oldroyd-B fluid. The basic idea of the pIB method is the splitting of an immersed boundary, which here is a rigid body, notionally into two Lagrangian components: one is a massive component carrying all mass of the rigid body, and the other is massless. These two components are connected by a system of stiff springs with zero rest length. The massive component has no direct interaction with the surrounding fluid and behaves as though in a vacuum, following the dynamics of a rigid body, in which the acting forces and torques are generated from the system of stiff springs that connects the two Lagrangian components. The massless component interacts with the surrounding Oldroyd-B fluid: it moves at the local fluid velocity and exerts force locally on the fluid. We verify the pIB method combined with Oldroyd-B fluid model by investigating the effects of the wall and elasticity of the fluid on the lateral position of a circular particle falling under the influence of gravity and by studying convergence of the numerical solutions. We also simulate the interaction between multiple circular particles and the surrounding Oldroyd-B fluid and compare the dynamics of the particles in various flow conditions.