Nowadays expert systems have been used in different fields. They must be able to operate as quickly and efficiently as possible. So, they need optimization mechanism in their different parts and optimization is a critical part of almost all expert systems. Because of difficulties in real world problems, traditional optimization techniques commonly cannot solve them. Therefore, stochastic algorithms are used to do the optimization in expert systems. Particle swarm optimization (PSO) is one the most famous stochastic optimization algorithms. But this algorithm has some difficulties like losing diversity, premature convergence, trapping in local optimums and imbalance between exploration and exploitation. To overcome these drawbacks, inspired by holonic organization in multi agent systems, a new hierarchical multi group structure for PSO is presented in this paper. Considering the particles in PSO as simple agents, PSO is a kind of multi agent system. Existence of different facilities and organizations in multi agent systems and their great impact on performance encouraged us to use them. So, inspired by holonic multi agent systems, a new structure for PSO is presented. This work has been done for the first time in the literature. Meanwhile, to promote exploration and exploitation ability of proposed structure and create a suitable balance between them, different tasks are assigned to different groups of this structure. So, a holonic PSO with different task allocations (HPSO-DTA) is created. It provides the opportunity to employ all aspects for empowering PSO including parameter settings, neighborhood topologies and learning strategies to enhance the ability of it unlike other versions of PSO that use only one of these aspects to improve their solutions. This structure provides a lot of advantages for PSO. It is a new topological structure that improves the performance of PSO. It provides several leaders with efficient information to guide the particles in the search space. Also, it helps to control suitable information flow between groups and particles in order to preserve diversity and prevent from trapping in local optimums. Meanwhile, with assigning different tasks to different groups of proposed structure, an appropriate balance between exploration and exploitation is created to enhance the performance of the algorithm. In each group, based on its assigned task, particles use different parameters settings, different dynamic neighborhood topologies and different learning strategies which are proposed in this paper to enhance the performance of algorithm. A set of thirty four benchmark functions are used to evaluate the performance of proposed structure. Proposed algorithm is compared with a set of well-known PSO algorithms that their efficiency have been proved. Experimental results and comparative analysis demonstrate good performance of HPSO-DTA compared to other algorithms. Its solution accuracy, convergence speed and robustness is completely appropriate especially in more complicated benchmarks.