Haptic feedback systems are systems that exert a desired force, to be experienced by the user, to recreate a physical interaction. This type of systems can increase the realism of the interaction with objects that are not in the direct area of influence of the user. The same applies to the interaction with virtual objects. The fundamental problem in the control of these systems is how to maximize the transparency of the system, and related to that the realism of the interaction as perceived by the user, while guaranteeing stability of the interaction under all possible operating conditions. The stability analysis is complicated due to the presence of a closed loop, which can contain multiple unknown, non-linear, and time-varying elements, e.g. the user, a physical environment, and time delays in a possible communication channel. Furthermore, the control algorithm is executed on a discrete medium which, depending on the algorithm, parameter settings, and remaining elements, can have a significant influence on the stability of the interaction. In this thesis a solution is sought to all these factors by means of energy- and portbased reasoning. For both applications an algorithm is derived that is based on the energy exchange between the physical world and the system. By enforcing energy neutrality, in other words passivity, of this exchange a stable interaction is guaranteed. The algorithm for the interaction with virtual environments provides a passive coupling between the physical system and the discrete system. The dynamic behavior of the virtual object is computed with an energy-based integration method. Each iteration the algorithm evaluates the energy that is present in the system. A redistribution of that energy over the energy storing elements is computed based on the model that describes the dynamic behavior of the physical object. This algorithm ensures stability independent of the sample frequency of the algorithm, but adapts the realism of the interaction in such a way that passivity is maintained. The second algorithm divides the control objectives for the passive interaction with physical objects in remote environments by means of a telemanipulation system into two layers that are placed in a hierarchical order. The top layer, the Transparency-layer, contains an arbitrary control algorithm that provides the desired measure of transparency. The Passivity-layer contains an algorithm that guarantees passivity of the interaction and when necessary adapts the desired forces computed by the Transparency-layer to maintain passivity. The implementation of this algorithm also works in the presence of communication delays, by e.g. physical distance, between both locations that are connected by the system, the user and remote environment, respectively.