Fully homomorphic encryption (FHE) provides a promising solution for future computing needs by allowing privacy-preserving computation. However, its practical use has been limited by the huge latency overhead it incurs while computing. This is primarily due to the huge size of encrypted data and intermediate processing required to compute on it. In this paper, we present insights into the benefits of accelerating FHE with processing in-memory (PIM). PIM is an excellent match for the FHE since it provides extensive parallelism, in-situ operations, and bit-level granularity. We present FHE-PIM, which implements basic polynomial primitives with PIM and uses them to accelerate key FHE operations in memory. This can significantly make the time-consuming procedure of FHE bootstrapping faster in memory. We compare the speedup of FHE-PIM for various FHE operations with their CPU implementations. FHE-PIM can achieve an estimated average throughput improvement of 88 as compared to CPU for FHE arithmetic operations.